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Institutional researchers and assessment 
professionals in higher education are living in interesting 
and challenging times, one might say in times of crisis. 
In the post-Reagan era government and its agencies 
and public professionals have come under fire for being 
ineffective and reluctant to reform and protected as special 
interests (House, 1993). As federal and state coffers have 
shrunk in a time of rising costs of higher education, 
university budgets are tightening while at the same time 
calls for accountability are increasing (U.S. Department 
of Education, 2006). The discourse of higher education 
quality has changed to focus on “return on investment” as 
the criteria for college and university success. Free market 
economics take the spotlight off of private enterprises but 
shift attention of policy makers to public organizations 
and as House (1993) asserts, “As a consequence, higher 
education.. .is blamed for social and economic problems 
that originate elsewhere, such as in the economic 
structure itself’ (p. 59). As a consequence, Terenzini 
(2009) asserts that assessment of student outcomes 
originating from external mandates is here to stay but 
that there are significant “conceptual, administrative, 
political and methodological issues” (p. 45) making 
such initiatives problematic. As a result, institutional 
researchers are caught in a conflicting press for efficiency, 
accountability and excellence. The research on the role of 
IR professionals reflects more emphasis on “accountability 
and performance” and “outcomes assessment” among 
other job tasks (Volkwein, 2008, p. 9). Here I attempt 
to offer perspective related to the “issues intelligence” 
as well as the “contextual intelligence” that make up the 
“organizational intelligence” of IR (Volkwein, 2008, p. 5; 
see also Terenzini, 1999). 

As in other social arenas the use of the term 
crisis always requires close inspection. Crises are socially 
constructed and they offer an opportunity to examine 
what an institution is doing and how it is conducting its 
operations. When a crisis is declared, it stands to reason 
that we should ask who is defining the crisis. Is the crisis 
defined in such a way as to determine new solutions 
and new patterns of operating? Are the calls for greater 
accountability, evaluation, and assessment motivated 
by forces to improve or discipline higher education? In 
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effect, times of crisis are good times to take a step back 
from the fray of day-to-day demands of standard operating 
procedures and examine what we are doing and why. 
This applies to evaluation and assessment professionals 
in institutional research and assessment offices within 
colleges and universities. We have to ask ourselves: Are 
our evaluation activities merely tools for “informing 
and legitimizing the unpopular steps that government 
must take, which often means budget cutting”? Are our 
evaluation activities merely lending scientific authority 
to questionable political decisions? “When professionals 
work in bureaucracies, their autonomy is often challenged. 
So professional versus bureaucratic interests is a central 
conflict. The conflict is increasingly manifested in the 

“Are our evaluation activities merely 
lending scientific authority to 
questionable political decisions?” 

higher education system, a stronghold of professional 
knowledge and legitimation. Governments have curtailed 
funding and increased their control over universities” 
(House, 1993, p. 53). Do productivity and efficiency 
trump the needs of our clients? To what degree should 
institutional researchers follow government policy and tc 
what degree must they follow the standards of their own 
profession? Ultimately these are matters of professional 
ethics and social justice” (House, 1993, p. 55). 

The field of evaluation itself exists in a political 
and social context and is at a crossroads of sorts. Writing 
in the 1990s, House’s (1993) words seem prophetic: 

Exactly in what form evaluation develops 
depends on how modern market societies 
develop. If these societies become more 
authoritarian, a distinct possibility in reaction 
to managing turbulent societies and sluggish 
economies, evaluation could be used for 
repressive purposes. On the other hand, 
if modern market societies become less 
ideological and more willing to consider new 
social possibilities, then evaluation could 
become more useful. .. .Being involved with 


RESEARCH ir PRACTICE IN ASSESSMENT 


government programs means that evaluation 
is always connected to ideological and political 
issues (pp. 28-29). 

The attacks on colleges and universities, “inspired 
by the ideology of the free market” and directed at 
the content as well as the costs, of higher education” 
(House, 1993, p. 59) are not solely from external bodies 
such as business interests, commissions, accrediting 
organizations, think tanks, or the media, they come from 
within the academy as well. When colleges and universities 
are criticized by researchers, working with foundation 
support, putatively because undergraduates don’t learn 
much as a result of attending our universities (Arum & 
Roksa, 2011), the stakes go up and the press for assessment 
goes up as well. “Measures imply a need for action” 
(Stone, 2002, p. 167). We get forced into an efficiency and 
accountability frame with other frames such as equity and 
quality pushed from the discussion. The tendency is to 
become myopic in our vision and practice of assessment. 
In 1993, House wrote: 

... (T)raditional autonomy that higher 
education institutions have enjoyed is coming 
to an end. The question is not whether we 
should have accountability, but rather what 
kind of accountability and evaluation we can 
have that will protect the vital internal processes 
of research and teaching that are essential to 
the improvement of society, and that will help 
protect higher education institutions against 
the economic and ideological assaults that are 
certain to come (pp.71-72). 

Many colleges and universities and their 
accrediting bodies are currently undergoing transitions 
brought on by changes in the economy and the political 
landscape with more and more universities acting hke 
for-profit enterprises. The accreditation process has 
shifted from one focused on self-study primarily based on 
inputs to the educational process, to an evidence-based, 
external accountability-oriented, outcomes-based model 
which focuses significantly on student learning outcome 
measures and assessment. The pressures of the new 
economy bring new challenges to institutional research 
and assessment offices as they conduct traditional duties of 

“...there may be a tendency to 
measure types of student outcomes 
that are easily measurable but not 
very meaningful.” 

program review, assessing faculty and student outcomes for 
accreditation and other accountability-related tasks. There 
is always the danger that evaluation, which has always been 
part of the legitimating functions of government, may 


devolve into mere institutional impression management 
rather than a useful tool for formative improvement. 

Will evaluation become a mere tool of “scientifically 
informing, legitimizing, and controlling” (House, 1993, 
p. 33)? 

There is a tension between public and professional 
accountability in higher education. Institutional 
researchers and evaluators are often in “conflicting and 
ambivalent circumstances” balancing demands of state 
or public accountability, professional accountability 
and consumer accountability (House, 1993, pp. 34-35). 
Institutional researchers are often caught in the middle of 
varying definitions of quality held by faculty, accreditors, 
students, parents, government officials, and trustees” 
(Volkwein, 2010, p. 10; see also Terenzini, 2010). 
Changes in the current era of neoliberal ideologies may 
encourage tendencies to ignore tried and true methods of 
university program review that include the perspectives 
of faculty members in the process. Or there may be a 
tendency to measure types of student outcomes that are 
easily measurable but not very meaningful. Are evaluators’ 
loyalties to central or local concerns or to political or 
professional authority? 

Evaluation and assessment are traditionally 
approached as a highly technical-rational field dominated 
by assumptions of neutrality, objectivity, rationality and 
technical details of psychometrics. Rarely do we stop 
to examine the assumptions of our job and why we are 
doing it. In addition, in college and university settings, 
assessment is part of a top-down institutional framework 
where questioning basic assumptions is not rewarded and 
is often times counter-productive to expectations of job 
performance and persistence within the bureaucracy. 
However, scholars of assessment understand that the field 
is embedded in a social, political, and historical context 
that shapes how we practice. We are part of a profession 
with professional goals and responsibilities. This is 
discernable from the agendas of professional groups such 
as the Virginia Assessment Group’s agenda for its 2011 
annual meeting which lists among other goals: 

• Engaging faculty in assessment policy and 
decision-making 

• Developing leadership in assessment across 
campus 

• Data quality versus quantity 

Numbers and Data Quality 

Institutional researchers attend primarily to 
quantitative indicators, survey data, assessment scores, 
performance data, and financial data. Most assessments 
and measures rely on numbers. Numbers are seemingly 
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technical, neutral, and rational but upon inspection, 
politics and interpretation are behind mere numbers. 
Numbers appear to have some cultural cache as the 
language of science. As Stone (2002) writes in her book 
Policy Paradox: 

“Numbers.. .are measures of human activities, 
made by human beings, and intended to influence human 
behavior. They are subject to conscious and unconscious 
manipulation by the people being measured, the people 
making the measurements, and the people who interpret 
and use measures made by others” (p. 177). Stone explains 
why counting is political not technical: 

• Counting requires decisions about 
categorizing, about what (or whom) to include 
and exclude. 

• Measuring any phenomenon implicitly 
creates norms about how much is too little, 
too much, or just right. 

• Numbers can be ambiguous, and so leave 
room for political struggles to control their 
interpretation. 

• Numbers are used to tell stories, such as 
stories of decline (“we are approaching a 
crisis”). 

• Numbers can create the illusion that a very 
complex and ambiguous phenomenon is 
simple, countable, and precisely defined. 

• Numbers can create political communities 
out of people who share some trait that has 
been counted. 

• Counting can aid negotiation and 
compromise, by making intangible qualities 
seem divisible. 

• Numbers, by seeming to be so precise, help 
bolster the authority of those who count. 

(2002, p. 176) 

She goes on to warn us about reactivity in 
measurement because measures always carry “implicit 
norms.” We attend to what we measure. When we have 
a hammer, everything becomes a nail. When we have the 
light of some off-the-shelf- assessment, we look for our 
lost keys there. Statistics are often used by those wishing 
to begin a reform effort (e.g., accountability in higher 
education), to show how bad things have become in order 
to support a narrative of decline. Numbers never stand on 
their own; they are always interpreted as part of a story 
line. Measurement exerts a strong force over behavior, and 
not always in a positive manner. Raising the level stakes of 
standardized tests means teachers will teach to the test or 
worse cheat on the test. The statistics and data we collect 
are collected in the interests of the organization and are 
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therefore often inseparable from the goals and agendas 
of those organizations. Stone (2002) laments: “Because 
policy measurement is always linked to benefits and 
penalties, the measured try to influence the measurers, 
occasionally with outright bribes, but more often with 
pleading, cajoling, and selective disclosure” (p. 182). 

According to Stone (2002), so much of what we 

“Numbers...are measures of human activities, 
made by human beings, and intended to 
influence human behavior. They are subject 
to conscious and unconscious manipulation 
by the people being measured, the people 
making the measurements, and the people 
who interpret and use measures made by 
others.” 

count depends on the categories we have constructed to 
count around. Category construction is a qualitative, 
interpretive, and ultimately a political activity. The 
discretion used in determining if one behavior fits in this or 
that category influences the numbers we produce, yet we 
rarely stop to question the category construction. Program 
costs can include just the cost of the program or it might 
also include the cost to design the program. The selection 
of measures almost always indicates some preferred 
solution to a problem. For instance, a foundation may 
be supporting research about the lack of student learning 
in higher education and may choose measurements that 
suggest a particular solution that positions the foundation 
to obtain more funding for its program of change and 
improvement. 

In the current climate of defining higher 
education policy problems, measurement plays a strategic 
role. Critics assert the policy problem is that colleges and 
universities are inefficient and do not produce learning. 
Numbers constructed by researchers that purport to show 
that undergraduates do not learning much in their first two 
years of college (Arum & Roksa, 2011) are used to support 
that a policy problem is growing. Measurement is always 
tied to a purpose: accountability, program improvement, 
cost efficiency, etc. Stone (2002) asserts: “Numerals hide 
all the difficult choices that go into a count” (p. 177). 

Volkwein (2008) proposes a typology of four 
possible IR purposes and roles. These vary by the purposes 
and audiences (formative/internal for improvement vs. 
summative and external for accountability) and whether 
the organizational role and culture emphasizes the 
administrative/institutional vs. academic/professional 
roles. If the role is one of administrative/institutional 
the job becomes one of describing and gathering facts 
about the institution (under the formative purpose) or to 
perform the role of spin-doctor in institutional impression 
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management. If IR departments and individuals take on 
the academic and professional role, the job becomes one 
of analyzing alternatives as the IR takes on the role policy 
analysis (under the formative purpose) and IR as scholar 
and researcher providing impartial evidence ofeffectiveness 
(under the summative purpose). Given the external 
conditions facing higher education in a time of neo-liberal 
ideologies, the commercialization ofhigher education, and 
increased political clamoring for accountability, I want to 
argue that institutional researchers engage the professional 
identity in terms of organizational roles (policy analysts and 
scholar/researcher) and that they think carefully about the 
use of numbers as they face the challenges of the new era 
in higher education. In the context of the massive changes 
in society that are having such a significant impact on 
higher education at this historical moment, we must ask: 
at what point does “thoughtful noncompliance” (Stein, 
2004) enter into the thinking of institutional researchers? 
How will we use the authority of numbers? 
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